OPEN-SOURCE KNOWLEDGE DISTILLATION TOOLKIT

Copied to clipboard

Challenge:	Large pre-trained language models have hundreds of millions of parameters and take several gigabytes of memory to train and inference.
Approach:	They propose an open-source knowledge distillation toolkit designed for natural language processing that provides a set of predefined distillation methods and can be extended with custom code.
Outcome:	The proposed method is comparable with or even higher than the public distilled BERT models with similar numbers of parameters.

Papers with open-source knowledge distillation toolkit